Artificial intelligence-based colonoscopic image diagnosis assisting system and method

ABSTRACT

A system and method for assisting colonoscopic image diagnosis based on artificial intelligence include a processor, which is configured to analyze each video frame of a colonoscopic image using at least one medical image analysis algorithm and detects a finding suspected of being a lesion in the video frame. The processor calculates the coordinates of the location of the finding suspected of being a lesion. The processor generates display information, including whether the finding suspected of being a lesion is present and the coordinates of the location of the finding suspected of being a lesion.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority from Korean Application No.10-2021-0035619 filed on Mar. 19, 2021, which is incorporated herein byreference in its entirety.

TECHNICAL FIELD

The present invention relates to a medical image diagnosis assistingapparatus and method using an automated system. More particularly, thepresent invention relates to an apparatus, system, and method forassisting the diagnosis of a colonoscopic image during colonoscopy byusing an artificial intelligence-based medical image analysis algorithmand also assisting the reduction of the risk of missing lesions.

The present invention was derived from the research conducted as part ofthe SW Computing Industry Fundamental Technology Development Projectsponsored by the Korean Ministry of Science and ICT and the Institute ofInformation & Communications Technology Planning & Evaluation [ProjectSerial No.: 1711116343; Project No.: 2018-0-00861-003; and Project Name:Intelligent SW Technology Development for Medical Data Analysis].

BACKGROUND ART

Endoscopic diagnosis is a medical practice that is considerablyfrequently performed for the purpose of regular medical examination.There is a demand for a technology that processes real-time images uponendoscopic diagnosis to preprocess them so that an expert can easilyidentify a lesion at a medical site. Recently, U.S. Patent ApplicationPublication No. US 2018/0253839 entitled “A System and Method forDetection of Suspicious Tissue Regions in an Endoscopic Procedure”introduced a technology that performed a preprocessing process ofremoving noise from an image frame and performed a noise removalpreprocessing process and a computer-aided diagnosis (CAD) process inparallel, thereby providing real-time diagnosis assistance displayinformation.

In this technology, the accuracy and reliability of a CAD module arerecognized as significantly important factors.

Technologies for segmenting or detecting objects in an image orclassifying objects in an image are used for various purposes in imageprocessing. In a medical image, objects in the image are segmented,detected, and classified based on the brightness or intensity values ofthe image, in which case each of the objects may be an organ of thehuman body, or a lesion.

Recently, the introduction of deep learning and a convolutional neuralnetwork (CNN) as artificial neural networks into the automation of animage processing process has dramatically improved the performance of anautomated image processing process.

However, on the other hand, the insides of recent artificial neuralnetworks, such as deep learning and a CNN, approximate black boxes, andthus there is reluctance for a user to fully accept and adopt them evenwhen acquired results are excellent. In particular, reluctance toartificial neural networks stands out as being more important in themedical imaging field in which human life is dealt with.

Under this background, research into explainable artificial intelligence(X-AI) has been attempted in the Defense Advanced Research and Planning(DARPA) of the U.S., etc. (seehttps://www.darpa.mil/program/explainable-artificial-intellig-ence).However, no visible results have yet been revealed.

In the medical field, as a technique for segmenting, detecting,classifying and diagnosing lesions having complex shapes, a techniquefor selectively applying a plurality of segmentation algorithms isdisclosed in International Publication No. WO2018/015414 entitled“Method and System for Artificial Intelligence Based Medical ImageSegmentation.”

In the related art document, a technique of comparing pre-trainedsegmentation algorithms and selecting at least one of the pre-trainedsegmentation algorithms is applied to the acquisition of a final resultof image segmentation.

However, descriptive information (explanation) about the criteria forthe selective application of the segmentation algorithms cannot bederived from the related art document, and thus a problem arises in thatit is difficult to increase a clinician's confidence in the clinicalusefulness of this segmentation technique.

Moreover, Korean Patent No. 10-1938992 entitled “CAD System and Methodfor Generating Description of Reason for Diagnosis” introduced atechnology that generated feature vectors by concatenating featureinformation extracted based on a DNN in order to derive groundinformation for the diagnosis of a lesion. However, in Korean Patent No.10-1938992, an artificial neural network derives feature information byitself, and no verification is made as to whether or not the extractedfeature information is clinically useful information. Accordingly, thereis little evidence that humans can recognize the above information as adescription of the diagnosis result of artificial neural networks.

A similar problem is still present in a medical image diagnosis processin that it is difficult to have clinical confidence in a process inwhich an artificial intelligence diagnosis system that operates like ablack box generates a result.

In particular, colonoscopy refers to an examination in which anendoscope is inserted into the human body through the anus and theinside of the colon and the distal end of the small intestine adjacentto the colon are observed. There is a video camera inside the device, sothat an operator (a doctor) can detect a polyp and/or abnormal tissueand remove the detected polyp and/or lesion to perform a biopsy oreliminate the polyp and/or tissue while monitoring the inside of thecolon with the camera installed in a tube. However, the early detectionof polyps is considerably important because polyps can develop intocancer later if they are overlooked during an examination.

During colonoscopy, lesions are often overlooked in the course ofexamination in a narrow path due to causes such as the resolution of animage, a view in which a user does not intuitively perceive athree-dimensional effect, a blind spot, user error, or the like. Inparticular, in group health checkups, doctors often exhibit signs offatigue due to continuous and repetitive procedures, so that lesions maynot be sufficiently detected. Therefore, human error may be combinedwith the cause in terms of the characteristics of images and cause themission of lesions, which may negatively affect the medical results ofexaminations.

SUMMARY

Recently, efforts have been made to improve the performance of imagesegmentation, object detection, and object classification techniques byapplying deep learning-based artificial intelligence techniques.However, in the case of deep learning-based artificial intelligence, thefact that there is a black box that prevents a user from determiningwhether or not a result provided from an operation accidentally exhibitshigh performance and whether or not a determination process appropriatefor a corresponding task has been performed limits the applicability ofthe deep learning-based artificial intelligence.

In contrast, the use of rule-based training or learning, which is easyto explain, is limited in that better performance cannot be achievedthan deep learning. Accordingly, research into deep learning-basedartificial intelligence that can provide descriptive information(explanation) while having improved performance is being activelyconducted. In the practical application of image processing using anartificial neural network, descriptive information about the basis ofdiagnosis and classification is required particularly in the medicalimaging field. However, descriptive information cannot be derived fromthe related art.

Even in the above-described related art document (InternationalPublication No. WO2018/015414), it is not possible to derive descriptiveinformation (explanation) on factors that affect the improvement offinal segmentation performance, and there is no way to verify thatclinically significant feedback has been actually and appropriatelyapplied to the deep learning system even when a clinician provides theclinically significant feedback.

An object of the present invention is to provide evaluation scores,including confidence and accuracy scores, for a plurality of medicalimage diagnosis algorithms in a process in which a user diagnoses amedical image, thereby improving the accuracy of a medical imagediagnosis result obtained by the user.

An object of the present invention is to provide recommended informationas descriptive information in a process in which a user derives a finaldiagnosis result by using artificial intelligence medical imagediagnosis algorithms, and to allow the user to provide information aboutthe clinical usefulness of the medical image diagnosis algorithms asquantified information.

An object of the present invention is to generate and provide anoptimized combination of a plurality of artificial intelligence medicalimage diagnosis results as display information for each real-time imageframe.

An object of the present invention is to provide an optimizedcombination of a plurality of artificial intelligence medical imagediagnosis results capable of efficiently displaying diagnosis resultsthat are likely to be acquired, are likely to be overlooked, or have ahigh level of risk in a current image frame.

An object of the present invention is to provide a user interface anddiagnosis computing system that automatically detect and presentdiagnosis results that are likely to be acquired, are likely to beoverlooked, or have a high level of risk in a current image frame, sothat medical staff can check and review the diagnosis results in realtime during an endoscopy.

An object of the present invention is to train on polyps, etc., whichmay be missed by a user, based on artificial intelligence medical imagediagnosis results for each real-time video frame of a colonoscopic imagevia an artificial intelligence algorithm and then apply the results ofthe training to an artificial intelligence diagnosis assisting system,thereby increasing work efficiency and diagnostic accuracy.

In general, it is known that an increase in the early detection rate ofpolyps has a close correlation with a decrease in the incidence ofcancer. Accordingly, an object of the present invention is to reduce theincidence of colorectal cancer by increasing lesion detection rate andalso eliminating colorectal cancer risk factors in their early stages.Furthermore, an object of the present invention is to contribute toreducing the causes of colorectal cancer and also reducing the frequencyof examinations by enabling doctors to find and treat more lesions thanbefore. The present invention utilizes polyps having various sizesand/or shapes, found and verified in real situations, together with thelocations at which they are found as training data for artificialintelligence/an artificial neural network. An object of the presentinvention is to increase polyp finding/detection rate by automaticallyfinding a polyp having a considerably small size, which is easy tooverlook with the human eye, and a polyp located in a blind spot byusing the artificial intelligence/artificial neural network trainedusing the above training data and also to lower the risk of developingcolon cancer via the above increase in polyp finding/detection rate.

An object of the present invention is to automatically detect apolyp/lesion that may easily be missed by a user during colonoscopy andpresent the location of the polyp/lesion in a colorectal path (acolonoscopic/colonoscopy path), so that the user may easily check thepolyp/lesion in real time during colonoscopy and even a report adaptedto enable other examiners to check it later may be generated through asimple operation.

In a diagnosis assisting technology for colonoscopic images, which is anapplication target of the present invention, when polyps and/or lesionshaving various sizes, shapes and locations are located together with thecolon wall or folds in the colon, it is easy to miss polyps and/orlesions when the polyps and/or lesions are not different in color fromsurrounding tissues and have small sizes. Accordingly, an object of thepresent invention is to provide a method that may further improve polypdetection rate in the colon by detecting various lesions in real timevia artificial intelligence and also to help other examiners to checkpolyp candidates again later by providing the locations of the polyps ina colonoscopic/colonoscopy path.

According to an aspect of the present invention, there is provided acolonoscopic image diagnosis assisting system including a computingsystem, wherein the computing system includes a receiving interface;memory or a database; a processor; and a user display. The receivinginterface is configured to receive a medical image (a colonoscopicimage) as the medical image, and the memory or a database is configuredto store at least one medical image analysis algorithm having a functionof analyzing the medical image (the colonoscopic image).

The processor is configured to analyze each video frame of thecolonoscopic image using the at least one medical image analysisalgorithm, and to detect a finding suspected of being a lesion in thevideo frame; the processor is further configured to calculate thecoordinates of the location of the finding suspected of being a lesion;and the processor is further configured to generate display information,including whether the finding suspected of being a lesion is present andthe coordinates of the location of the finding suspected of being alesion.

The user display is configured to display the finding suspected of beinga lesion so that it is visually distinguished in the video frame basedon the display information and display the coordinates of the locationof the finding suspected of being a lesion so that they are visuallyassociated with the finding suspected of being a lesion.

The processor may be further configured to calculate the location of thefinding suspected of being a lesion in a colonoscopic path; and theprocessor may be further configured to generate display information,including whether the finding suspected of being a lesion is present,the coordinates of the location of the finding suspected of being alesion, and the location of the finding suspected of being a lesion inthe colonoscopic path.

The user display may be further configured to display the location ofthe finding suspected of being a lesion in the colonoscopic path so thatit is visually associated with the finding suspected of being a lesionbased on the display information.

The processor may be further configured to track the location of thevideo frame, indicative of a current examination region, in thecolonoscopic path; and the processor may be further configured tocalculate the location of the finding suspected of being a lesion in thecolonoscopic path based on the location of the video frame in thecolonoscopic path and the coordinates of the location of the findingsuspected of being a lesion.

The processor may be further configured to calculate the location of thefinding suspected of being a lesion in the colonoscopic path based on apre-examination medical image including the three-dimensional anatomicalstructure of a patient to be examined.

The artificial intelligence-based medical image (colonoscopic image)analysis algorithm may be trained using a label, including an indicationof a lesion detected for each video frame, the coordinates of thelocation of the lesion in the video frame, and the location of thelesion in a colonoscopic path, together with each video frame astraining data. Accordingly, the processor may calculate the location ofthe finding suspected of being a lesion in the video frame in thecolonoscopic path by using the medical image analysis algorithm.

The processor may be further configured to extract context-baseddiagnosis requirements corresponding to the characteristics of the videoframe in the medical image by analyzing the video frame of the medicalimage. The processor may be further configured to select a diagnosisapplication algorithm to perform diagnosis on the video frame from amonga plurality of diagnosis application algorithm candidates based on thecontext-based diagnosis requirements.

In an embodiment of the present invention, the receiving interface maybe further configured to receive at least one colonoscopic image from atleast one colonoscopic image acquisition module. In this case, theprocessor may be further configured to detect a finding suspected ofbeing a lesion for each video frame of the at least one colonoscopicimage by using the at least one medical image analysis algorithm. Theprocessor may be further configured to generate display information,including whether the finding suspected of being a lesion is present andthe coordinates of the location of the finding suspected of being alesion, for each video frame of the at least one colonoscopic image.

According to another aspect of the present invention, there is provideda colonoscopic image diagnosis assisting method that is performed by acolonoscopic image diagnosis assisting system including a processor anda user display and utilizes at least one medical image analysisalgorithm having a function of analyzing a colonoscopic image stored inmemory or a database inside the colonoscopic image diagnosis assistingsystem.

The colonoscopic image diagnosis assisting method includes: receiving acolonoscopic image; analyzing, by the processor, each video frame of thecolonoscopic image by using at least one medical image analysisalgorithm, and detecting, by the processor, a finding suspected of beinga lesion in the video frame; when the finding suspected of being alesion is present in the video frame, calculating, by the processor, thecoordinates of the location of the finding suspected of being a lesion;generating, by the processor, display information, including whether thefinding suspected of being a lesion is present and the coordinates ofthe location of the finding suspected of being a lesion; and, when thefinding suspected of being a lesion is present in the video frame,displaying, by the user display, the finding suspected of being a lesionso that it is visually distinguished in the video frame based on thedisplay information; and displaying, by the user display, thecoordinates of the location of the finding suspected of being a lesionso that they are visually associated with the finding suspected of beinga lesion.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other objects, features and advantages of the presentinvention will be more clearly understood from the following detaileddescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a view showing an artificial intelligence-based colonoscopicimage diagnosis assisting system according to an embodiment of thepresent invention and a workflow using the same;

FIG. 2 is a view showing the workflow of an artificialintelligence-based colonoscopic image diagnosis assisting systemaccording to an embodiment of the present invention;

FIG. 3 is a view showing a process of generating training data throughthe image processing/analysis of a colonoscopic image according to anembodiment of the present invention;

FIG. 4 is a diagram showing a colonoscopic image diagnosis assistingsystem having a multi-client structure and peripheral devices accordingto an embodiment of the present invention;

FIG. 5 is a diagram showing a colonoscopic image diagnosis assistingsystem having a single client structure and peripheral devices accordingto an embodiment of the present invention;

FIG. 6 is a view showing a colonoscopic image diagnosis assisting systemand a workflow using the same according to an embodiment of the presentinvention; and

FIG. 7 is a view showing a colonoscopic image diagnosis assisting systemand a workflow using the same according to an embodiment of the presentinvention.

DETAILED DESCRIPTION

Other objects and features of the present invention in addition to theabove-described objects will be apparent from the following descriptionof embodiments to be given with reference to the accompanying drawings.

Embodiments of the present invention will be described in detail belowwith reference to the accompanying drawings. In the followingdescription, when it is determined that a detailed description of aknown component or function may unnecessarily make the gist of thepresent invention obscure, it will be omitted.

When recently rapidly developed deep learning/CNN-based artificialneural network technology is applied to the imaging field, it may beused to identify visual elements that are difficult to identify with theunaided human eye. The application of this technology is expected toexpand to various fields such as security, medical imaging, andnon-destructive inspection.

For example, in the medical imaging field, there are cases where cancertissue is not immediately diagnosed as cancer during a biopsy but isdiagnosed as cancer after being tracked and monitored from apathological point of view. Although it is difficult for the human eyeto confirm whether or not corresponding cells are cancer in a medicalimage, there is an expectation that the artificial neural networktechnology can provide a more accurate prediction than the human eye.

However, although the artificial neural network technology can yieldbetter prediction/classification/diagnosis results than the human eye insome studies, there is a lack of descriptive information aboutprediction/classification/diagnosis results acquired through theapplication of the artificial neural network technology, and thus aproblem arises in that it is difficult to accept and adopt the aboveresults in the medical field.

The present invention has been conceived from the intention to improvethe performance of the classifying/predicting objects in an image, whichare difficult to classify with the unaided human eye, through theapplication of the artificial neural network technology. Furthermore,even in order to improve the classification/prediction performance ofthe artificial neural network technology, it is significantly importantto acquire descriptive information about the internal operation thatreaches the generation of a final diagnosis result based on theclassification/prediction processes of the artificial neural networktechnology.

The present invention may present the performance indicators andclinical usefulness of a plurality of medical image diagnosis algorithmsbased on artificial neural networks as quantified indicators. As aresult, it is possible to provide descriptive information about aprocess of acquiring a final diagnosis result based on theclassification/prediction processes of the artificial neural network,and it is also possible to provide a reference for the determination ofwhether or not a human user can adopt theclassification/prediction/diagnosis results of an artificial neuralnetwork.

When the artificial neural networks of the related arts are applied tothe diagnosis of medical images, they are overfitted only for giventasks, so that statistical accuracy is high but accuracy is low in someclinically important diagnostic points. Many neural networks of therelated art are in such a situation, and thus there occur frequent caseswhere it is difficult for clinicians to have confidence in the diagnosisresults for medical images to which the artificial neural networks areapplied. This risk is more obvious in that IBM's Watson Solution, awell-known artificial neural network, exhibits a problem in that it isoverfitted for patient race information included in learned data andthus it is significantly low in accuracy in the case of the dataset ofnew race patients.

Therefore, it is significantly important to provide a route throughwhich quantified indicators regarding whether or not clinicians willaccept these diagnosis results can be provided and clinicians canprovide direct feedback on the generation of the quantified indicatorswhile maximally utilizing the excellent analytical/diagnostic potentialof the artificial neural networks.

The aforementioned U.S. Patent Application Publication No. US2018/0253839 entitled “A System and Method for Detection of SuspiciousTissue Regions in an Endoscopic Procedure,” International PublicationNo. WO2018/015414 entitled “Method and System for ArtificialIntelligence Based Medical Image Segmentation,” and Korean Patent No.10-1938992 entitled “CAD System and Method for Generating Description ofReason for Diagnosis” disclose basic components for the artificialintelligent-based diagnosis of endoscopic images, i.e., an endoscopicimage acquisition module, an image capture and image processing module,and a transmission/reception interface (module) which transmits anacquired/captured endoscopic image to a computing system equipped withan analysis engine, and memory or a database in which an artificialintelligence/artificial neural network-based image analysisalgorithm/engine is stored.

In the present invention, a data storage means, a computation means, thebasic concept and structure of an artificial neural network, atransmission/reception interface for transferring input data (an image),etc. are required to implement an invention. However, detaileddescriptions of these basic elements may make the gist of the presentinvention obscure. Among the components of the present invention, theitems known to those of ordinary skill in the art prior to the filing ofthe present application will be described as parts of the components ofthe present invention in the present specification, if necessary.However, if it is determined that a fact obvious to those of ordinaryskill in the art may make the gist of the invention obscure, adescription thereof may be omitted.

In addition, descriptions of the items omitted therein may be replacedby providing notification that the items are known to those of ordinaryskill in the art via the related art documents, e.g., U.S. PatentApplication Publication No. US 2018/0253839 entitled “A System andMethod for Detection of Suspicious Tissue Regions in an EndoscopicProcedure,” International Publication No. WO2018/015414 entitled “Methodand System for Artificial Intelligence Based Medical ImageSegmentation,” and Korean Patent No. 10-1938992 entitled “CAD System andMethod for Generating Description of Reason for Diagnosis,” that arecited therein.

A medical image diagnosis assisting apparatus and method according toembodiments of the present invention will be described in detail belowwith reference to FIGS. 1 to 7.

FIG. 1 is a view showing an artificial intelligence-based colonoscopicimage diagnosis assisting system according to an embodiment of thepresent invention and a workflow using the same.

FIG. 2 is a view showing the workflow of an artificialintelligence-based colonoscopic image diagnosis assisting systemaccording to an embodiment of the present invention.

Referring to FIGS. 1 and 2 together, the colonoscopic image diagnosisassisting system according to the present embodiment includes acomputing system, and the computing system includes a receivinginterface; memory or a database; a processor; and a user display.

The colonoscopic image diagnosis assisting system according to thepresent embodiment may include one computing system, or may include aplurality of computing systems. In this case, the plurality of computingsystems may perform individual functions, and may perform the workflowof the colonoscopic image diagnosis assisting system in cooperation withother computing systems.

For example, one of the computing systems may be an artificialintelligence analysis server 110. The artificial intelligence analysisserver 110 may analyze and detect whether a polyp is present in eachframe by analyzing a medical image (a colonoscopic image), and maygenerate and output the results of the analysis and detection.

For example, the receiving interface (not shown) of the artificialintelligence analysis server 110 receives a medical image, and thememory or database of the artificial intelligence analysis server 110may store at least one medical image analysis algorithm 212 having afunction of diagnosing a medical image (a colonoscopic image). In thiscase, referring to FIGS. 1 and 2, the medical image transferred to theartificial intelligence analysis server 110 may be an image set that isthe result of image analysis 220 performed by an artificial intelligenceworkstation 120 and is image-processed and standardized such that it canbe compared with training input data.

The colonoscopic image acquisition module 132 may transfer acolonoscopic image, acquired in real time during endoscopy 232, to agateway 126 in a DICOM format. In this case, not only a real-time imageduring the endoscopy 232 but also an image of the colonoscopic imageacquisition module 132 during the endoscopy 232 may be stored in adatabase in a hospital. Thereafter, even when the image is inquired,viewed and/or checked by the same user or another user (see 234), thecolonoscopic image diagnosis assisting system of the present inventionmay be applied.

In this case, the colonoscopic image acquisition module 132 may converta non-DICOM image into a DICOM format and transmit it to the gateway126. The gateway 126 may convert a DICOM image into a JPEG format andtransmit it to the artificial intelligence workstation 120.

In another embodiment of the present invention, the colonoscopic imageacquisition module 132 may transfer a colonoscopic image, acquired inreal time, to the gateway 126 as a non-DICOM image, and the gateway 126may convert the non-DICOM image into a DICOM format and transfer a DICOMimage or JPEG/MPEG format image to the artificial intelligenceworkstation 120.

The artificial intelligence workstation 120 may also include a receivinginterface (not shown). The processor (not shown) of the artificialintelligence workstation 120 may perform image processing, includingcropping adapted to remove the black border portion of a colonoscopicimage and/or a captured image, rotation/tilting, and the correction ofimage brightness values, as part of the image analysis 220. Theprocessor of the artificial intelligence workstation 120 may transferthe medical image, converted into a standardized format as a result ofthe image analysis 220, to the artificial intelligence analysis server110. In this case, the standardized format may be a well-known format,such as PNG, JPEG, or MPEG, for the purpose of the easy application ofan artificial neural network.

The processor (not shown) of the artificial intelligence analysis server110 may analyze the video frame of the colonoscopic image (alreadyprocessed through the image analysis 220) using the at least one medicalimage analysis algorithm 212, and may detect a finding suspected ofbeing of lesion/colon, whether the finding suspected of being alesion/colon is present within the video frame. When the findingsuspected of being a lesion is present in the video frame, the processorcalculates the coordinates of the location of the finding suspected ofbeing a lesion, and generates an analysis result 214 including whetherthe finding suspected of being a lesion is present and the coordinatesof the location of the finding suspected of being a lesion. Theprocessor generates display information to be displayed together withthe colonoscopic image based on the analysis result 214.

The processor of the artificial intelligence analysis server 110 mayextract context-based diagnosis requirements corresponding to thecharacteristics of the video frame by analyzing the video frame of themedical image (see 216). In this case, the processor may select at leastone diagnosis application algorithm to perform the diagnosis/analysis ofthe video frame from among the plurality of medical image analysisalgorithms 212, i.e., diagnosis application algorithm candidates, basedon the context-based diagnosis requirements.

The user display 122 displays the analysis result 214 together with thecolonoscopic image so that a user can check it (see 222). In otherwords, when a finding suspected of being a lesion is present in thevideo frame, the user display 122 displays the finding suspected ofbeing a lesion so that it can be visually distinguished in the videoframe based on the display information, and also displays thecoordinates of the location of the finding suspected of being a lesionso that they can be visually associated with the finding suspected ofbeing a lesion. The user's final verification (see 224) process isperformed in such a manner that a user (a medical professional) acceptsor rejects the finding suspected of being a lesion displayed in thecurrent video frame.

The processor of the artificial intelligence analysis server 110 maycalculate the location of the finding suspected of being a lesion in acolonoscopic/colonoscopy path, and may generate display informationincluding whether the finding suspected of being a lesion is present,the coordinates of the location of the finding suspected of being alesion, and the location of the finding suspected of being a lesion inthe colonoscopic path. In this case, the processor may calculate thelocation of the finding suspected of being a lesion in the colonoscopicpath based on the information of a sensor on a colonoscopic deviceand/or the analysis result 214 of the artificial intelligence algorithm212.

The user display 122 may display the location of the finding suspectedof being a lesion in the colonoscopic path so that it is visuallyassociated with the finding suspected of being a lesion based on thedisplay information (see 222), with the result that a user can check it.

The processor of the artificial intelligent analysis server 110 maytrack the location of a video frame, indicative of a current examinationregion, in the colonoscopic path, and may calculate the location of thefinding suspected of being a lesion in the colonoscopic path based onthe location of the video frame in the colonoscopic path and thecoordinates of the location of the finding suspected of being a lesion.

The processor of the artificial intelligent analysis server 110 maycalculate the location of the finding suspected of being a lesion in thecolonoscopic path based on a pre-examination medical image including thethree-dimensional (3D) anatomical structure of a patient to be examined.

The analysis result 214 provided by the artificial intelligence analysisserver 110 may include the detected coordinates of the lesion/polyp, theprobability that the lesion/polyp is detected, and/or the location ofthe lesion/polyp in the colonoscopic path.

A user may finally verify the display information displayed togetherwith the endoscopic image (see 224), may accept or reject the findingsuspected of being a lesion, included in the display information, as alesion, and may, when the finding suspected of being a lesion isaccepted as a lesion, take subsequent actions for the lesion or preparea report in order to take subsequent actions later, thereby causingcolonoscopy to be terminated.

The artificial intelligence-based medical image (colonoscopic image)analysis algorithm 212 may be trained using a label, including anindication of a lesion detected for each video frame, the coordinates ofthe location of the lesion in the video frame, and the location of thelesion in a colonoscopic path, together with each video frame astraining data. Accordingly, the processor may calculate the location ofthe finding suspected of being a lesion in the video frame in thecolonoscopic path by using the medical image analysis algorithm 212, andmay provide it as the analysis result 214.

In an embodiment of the present invention, a main means for identifyinga current location in a path indicated by an endoscopic image may mainlydepend on learning and inference regarding endoscopic images.

In this case, when a current location in a colonoscopic path isidentified depending on learning and reasoning regarding endoscopicimages, the label of each endoscopic image used for learning may includeinformation about a location in an endoscopic (colonoscopic) path andinformation about a lesion detected/verified (an image actually verifiedthrough biopsy) separately for each frame.

In another embodiment of the present invention, learning and inferenceregarding endoscopic images is a main means for identifying a currentlocation in a colorectal path, and additionally a current location maybe more accurately identified by combining the main means with anadditional means for identifying the current location by estimating theprogress speed of frames of an endoscopic image through image analysis.

Furthermore, in general, it is difficult to take a CT image beforeendoscopy, so that it is necessary to identify a current location in acolorectal path by relying only on an endoscopic image. However, if itis possible to take a CT image before endoscopy, a current location maybe identified in association with a 3D model of an endoscopic target(the colon) reconstructed based on a CT image taken before endoscopy inanother embodiment of the present invention.

In this case, a CT image-based 3D model of the colon may be implementedin combination with virtual endoscopic imaging technology, whichcorresponds to the patent issued to the present applicant (Korean PatentNo. 10-1850385 or 10-1230871).

Furthermore, according to another embodiment of the present invention,when a current location in an endoscopic path within an endoscopicexamination target (the colon) is identified, correction (compensation)may be performed in association with an endoscope or a sensor installedin an endoscopic device (a sensor capable of detecting the length of anendoscope inserted into the human body) rather than relying solely onimage analysis.

According to an embodiment of the present invention, the receivinginterface of the artificial intelligence analysis server 110 may receiveat least one endoscopic image from at least one endoscopic imageacquisition module. In this case, the processor of the artificialintelligence analysis server 110 may detect whether a finding suspectedof being a lesion is present in each video frame of at least oneendoscopic image by using at least one medical image analysis algorithm212. The processor may generate display information, includinginformation about whether a finding suspected of being a lesion ispresent and the coordinates of the location of the finding suspected ofbeing a lesion, for each video frame of at least one colonoscopic image.

The display information may include whether a finding suspected of beinga lesion is present, the coordinates of the location of the findingsuspected of being a lesion (the coordinates of a location within acurrent video frame), and the location of the finding suspected of beinga lesion in a colonoscopic path.

The finding suspected of being a lesion is visualized to be visuallydistinguished from other parts in the video frame of the colonoscopicimage. In this case, the corresponding finding may be marked with avisualization element such as a marker/box, or may be highlighted.

Furthermore, information about the location and the probability that thelesion in question is actually a lesion (the probability inferred byartificial intelligence) are included in the display information so thatit can be visualized such that a user can intuitively understand theproximity or relevance of the finding suspected of being a lesion to thevisualization element.

The display information may include an endoscopic image frame, adiagnosis result selectively overlaid on the endoscopic image frame,information about a diagnosis application algorithm having generated thediagnosis result, and an evaluation score for the diagnosis applicationalgorithm.

Artificial intelligence diagnosis algorithms may be applied to diagnosisin the descending order of the evaluation score. However, in specialsituations, weights may be adaptively allocated to the detailed items ofthe evaluation score.

A medical image diagnosis assisting method according to anotherembodiment of the present invention is performed by a processor in adiagnosis assisting system (a computing system, e.g., the artificialintelligence analysis server 110) that assists the diagnosis of medicalimages, and is performed based on program instructions that are loadedinto the processor.

A colonoscopic image diagnosis assisting method according to anembodiment of the present invention is performed by the colonoscopicimage diagnosis assisting system (a computing system, e.g., theartificial intelligence analysis server 110) including the processor andthe user display 122, and may utilize the at least one medical imageanalysis algorithm 212 having a colonoscopic analysis function stored inthe memory or the database included in the colonoscopic image diagnosisassisting system.

The method of the present invention includes the steps of: receiving acolonoscopic image; analyzing, by the processor, each video frame of acolonoscopic image by using the at least one medical image analysisalgorithm 212, and detecting, by the processor, whether a findingsuspected of being a lesion is present in the video frame; calculating,by the processor, the coordinates of the location of a finding suspectedof being a lesion when the finding suspected of being a lesion ispresent in the video frame; generating, by the processor, displayinformation including whether a finding suspected of being a lesion ispresent and the coordinates of the location of the finding suspected ofbeing a lesion; when the finding suspected of being a lesion is presentin the video frame, displaying, by the user display, the findingsuspected of being a lesion on the video frame so that it is visuallydistinguished in the video frame based on the display information; anddisplaying, by the user display, the coordinates of the location of thefinding suspected of being a lesion so that they are visually associatedwith the finding suspected of being a lesion.

In this case, the method of the present invention may further includethe step of calculating, by the processor, the location of a findingsuspected of being a lesion in a colonoscopic path.

In the method of the present invention, the step of generating thedisplay information includes the step of generating, by the processor,display information, including whether a finding suspected of being alesion, the coordinates of the location of the finding suspected ofbeing a lesion, and the location of the finding suspected of being alesion in the colonoscopic path.

In the method of the present invention, the step of displaying, by theuser display, the coordinates of the location of the finding suspectedof being a lesion so that they are visually associated with the findingsuspected of being a lesion includes the step of displaying, by the userdisplay, the location of the finding suspected of being a lesion in thecolonoscopic path so that it is visually associated with the findingsuspected of being a lesion.

In the method of the present invention, the step of receiving thecolonoscopic image may include the step of receiving at least onecolonoscopic image from the at least one colonoscopic image acquisitionmodule.

In the method of the present invention, the step of detecting whether afinding suspected of being a lesion is present in the video frameincludes the step of detecting whether a finding suspected of being alesion is present in each video frame of the at least one endoscopicimage by using the at least one medical image analysis algorithm.

In the method of the present invention, the step of generating thedisplay information includes the step of generating display information,including information about whether a finding suspected of being alesion is present for each video frame of the at least one endoscopicimage and the coordinates of the location of the finding suspected ofbeing a lesion.

FIG. 3 is a view showing a process of generating training data throughthe image processing/analysis of a colonoscopic image according to anembodiment of the present invention.

Referring to FIG. 3, there is shown an embodiment of the raw image 310of a colonoscopic image obtained from a colonoscopy diagnosis apparatusaccording to an embodiment of the present invention. The raw image 310of the colonoscopic image includes a colonoscopic image part 320 of anexamination region, and display information.

In the process of training an artificial intelligence analysis algorithmfor colonoscopic images according to an embodiment of the presentinvention, training input data includes the following. The raw image 310of a colonoscopy image used as input data for training includes an imageincluding a black background having a size varying depending on theresolution supported by an image acquisition device (a colonoscopicimage acquisition module). In order to use only colonoscopic imageinformation, endoscopic image cropping is performed prior to training.Although the training input data may include the raw image 310, thecolonoscopic image part 320 and label information 330 obtained throughthe cropping are mainly used.

From the raw image 310, information about a patient, clinicalrequirements for the patient, verified diagnostic comments for acolonoscopic image frame, and a lesion detection result for thecolonoscopic image frame are extracted and generated as the labelinformation 330. In this case, the information about the patient andclinical requirements for the patient may be entirely extracted from theraw image 310, or may be supplemented by using in-hospital databasessuch as PACS, EMR, OCS, CIS, and HIS. In this case, the location in acolonoscopic path corresponding to the frame of the raw image 310 mayalso be extracted and included in the label information 330.

In a learning (training) stage, a location in a colonoscopic path may belearned along with information for the detection of a lesion (in thestate of being included in label information), and, finally, thecoordinate values where the lesion is located, the probability of beinga lesion, and the result of the location in the path may be learned.

In a process of performing inference for a real-time image afterlearning, the result value of the analysis is displayed on the userscreen by using a visualization element that can be visuallydistinguished. In this case, a process of extracting features from thecolonoscopic image part 320 in the artificial intelligence analysisserver 110 or the like may be performed by an artificial neuralnetwork-based image analysis process such as ResNet-50. At least onefeature for each layer extracted by ResNet-50 is provided to an FPN(feature pyramid network) along with a box probability value, includinga class probability value for whether a polyp in question is a polyp foreach layer and information about the location of the polyp, it isdetermined whether the polyp has been accurately detected by calculatingthe values, learning is performed, and, finally, the result of thecoordinate values where the polyp is located are obtained.

In order to further reduce a user's risk of missing, when a dangerousfinding is found, the user's attention may be called by using an alarmsound additionally. When the risk of mission is high due to the type ofdangerous finding, the probability that the dangerous finding is alesion, or the location of the dangerous finding in a blind spot in thefield of view, different alarm sounds may be used in respective cases inorder to further call the user's attention.

As a means for calling the user's attention, a bounding box including alesion/polyp risk region may be displayed on the user display 122, or aspecific message or a lesion/polyp risk region may be displayed via apop-up window. This alarm sound, bounding box, message, or pop-up windowmay be maintained for a specific amount of time (e.g., one second).

When training data is generated, the augmentation of data may beperformed in order to resolve overfitting attributable to a specificbias (the color, brightness, resolution, or tilting of an endoscopedevice) of the data. The augmentation of data may be achieved throughthe rotation/tilting, movement, symmetry, and correction of thecolor/brightness/resolution of image data.

In addition, as an example of a method for preventing overfitting, theremay be used various methods such as the weight regulation, dropoutaddition, and network capacity adjustment (reduction) of an artificialneural network included in the artificial intelligence analysis server110.

A process of generating the training data of FIG. 3 may be performed asa part of the image analysis 220 performed by the artificialintelligence workstation 120 of FIG. 1. However, in another embodimentof the present invention, the artificial intelligence analysis servermay perform all the functions of the artificial intelligenceworkstation, and the artificial intelligence analysis server may performthe image analysis 220 and the generation of training data.

Meanwhile, the image analysis/processing process such as croppingperformed in the learning stage may be performed as part of the imageanalysis (220) process even in the inference stage, so that theartificial intelligence analysis server 110 may receive a standardizedinput image to be compared with training data and apply the artificialneural network and/or the image analysis algorithm 212.

FIG. 4 is a diagram showing a colonoscopic image diagnosis assistingsystem having a multi-client structure and peripheral devices accordingto an embodiment of the present invention.

A first colonoscopic image acquisition module 432 may transfer acolonoscopic image, acquired in real time, to an artificial intelligenceworkstation 420 in real time, or may transfer a captured image of acolonoscopic image to the artificial intelligence workstation 420.

A second colonoscopic image acquisition module 434 may transfer acolonoscopic image, acquired in real time, to the artificialintelligence workstation 420 in real time, or may transfer a capturedimage of a colonoscopic image to the artificial intelligence workstation420.

In this case, the artificial intelligence workstation 420 may include aninput/reception interface module (not shown) configured to receive acolonoscopic image (or a captured image) received from the firstcolonoscopic image acquisition module 432 and the second colonoscopicimage acquisition module 434.

The artificial intelligence workstation 420 may transfer a video frameof a received/acquired colonoscopic image to an artificial intelligenceserver 410. In this case, the image transferred to the artificialintelligence server 410 may be transferred in a standardized JPEG orMPEG format. The artificial intelligence server 410 may also include aninput/reception interface module (not shown) configured to receive avideo frame image in a standardized format.

The artificial intelligence server 410 may detect and determine a lesionin a video frame/image by using an artificial intelligence algorithm (amedical image analysis algorithm) 412.

A processor (not shown) in the artificial intelligence server 410 mayinput given image data to the artificial intelligence algorithm 412, mayreceive the analysis result of the artificial intelligence algorithm412, and may control a data transfer process between memory or storage(not shown) having the artificial intelligence algorithm 412 storedtherein and the processor during the above process.

The output interface module (not shown) of the artificial intelligenceserver 410 may transfer the analysis result of the artificialintelligence algorithm 412 to the artificial intelligence workstation420. In this case, the transferred information may include whether afinding suspected of being a lesion is detected within a video frame,the coordinates at which the finding suspected of being a lesion isdetected, the probability that the finding suspected of being a lesionis a lesion, and the location of the finding suspected of being a lesionin a colorectal or colonoscopic path.

The artificial intelligence workstation 420 may display the analysisresult of the artificial intelligence algorithm 412 on a user display422. In this case, the information displayed on the user display 422 mayinclude whether a finding suspected of being a lesion is detected withina video frame of a currently displayed colonoscopic image, the findingsuspected of being a lesion that is visualized to be visuallydistinguished in the video frame (e.g., the finding suspected of being alesion that is highlighted, or the finding suspected of being a lesionthat is surrounded with a box), the coordinates of the location of thefinding suspected of being a lesion within the video frame, and thelocation of the finding suspected of being a lesion in a colorectal orcolonoscopic path.

In this case, a plurality of colonoscopic images may be simultaneouslydisplayed on the user display 422 in real time. Individual colonoscopicimage video frames may be displayed for respective windows on a screen.

In real-time examination, when a user immediately detects a findingsuspected of being a lesion on a colonoscopic image and takes action,there is no significant problem. In contrast, when a user misses afinding suspected of being a lesion in a real-time examination, it issubstantially impossible to re-diagnose the missing finding suspected ofbeing a lesion in the related arts.

Meanwhile, the present invention has advantages in that a captured videoframe may be re-checked by other users later, in that even when anendoscope has already advanced, a previous video frame may be recalledand the missing finding suspected of being a lesion may be re-diagnosed,and in that even after endoscopy has been finished, the coordinates ofthe location of a finding suspected of being a lesion within the videoframe and the location of the finding suspected of being a lesion in acolonoscopic path are provided together, and thus the location of thefinding suspected of being a lesion may be identified, so that follow-upactions may be taken.

Although FIG. 4 shows an embodiment in which the artificial intelligenceworkstation 420 and the artificial intelligence server 410 are separatedfrom each other for convenience of description, this is merely anembodiment of the present invention. However, it will be apparent tothose of ordinary skill in the art that according to another embodimentof the present invention, the artificial intelligence workstation 420and the artificial intelligence server 410 may be implemented to becombined together in a single computing system.

FIG. 5 is a diagram showing a colonoscopic image diagnosis assistingsystem having a single client structure and peripheral devices accordingto an embodiment of the present invention.

A colonoscopic image acquisition module 532 may transfer a colonoscopicimage, acquired in real time, to an artificial intelligence workstation520 in real time, or may transfer a captured image of a colonoscopicimage to the artificial intelligence workstation 520.

In this case, the artificial intelligence workstation 520 may include aninput/reception interface module (not shown) configured to receive acolonoscopic image (or a captured image) received from the colonoscopicimage acquisition module 532.

The artificial intelligence workstation 520 may detect and determine alesion in a video frame/image by using an artificial intelligencealgorithm (a medical image analysis algorithm) 512.

A processor (not shown) in the artificial intelligence workstation 520may input given image data to the artificial intelligence algorithm 512,may receive the analysis result of the artificial intelligence algorithm512, and may control a data transfer process between memory or storage(not shown) having the artificial intelligence algorithm 512 storedtherein and the processor during the above process.

The output interface module (not shown) of the artificial intelligenceworkstation 520 may generate the analysis result of the artificialintelligence algorithm 512 as display information, and may transfer thedisplay information to a user display 522. In this case, the transferredinformation may include whether a finding suspected of being a lesion isdetected within a video frame, the coordinates at which the findingsuspected of being a lesion is detected, the probability that thefinding suspected of being a lesion is a lesion, and the location of thefinding suspected of being a lesion in a colorectal or colonoscopicpath.

The artificial intelligence workstation 520 may display the analysisresult of the artificial intelligence algorithm 512 on a user display522. In this case, the information displayed on the user display 522 mayinclude whether a finding suspected of being a lesion is detected withina video frame of a currently displayed colonoscopic image, the findingsuspected of being a lesion that is visualized to be visuallydistinguished within the video frame (e.g., the finding suspected ofbeing a lesion that is highlighted, or the finding suspected of being alesion that is surrounded with a box), the coordinates of the locationof the finding suspected of being a lesion within the video frame, andthe location of the finding suspected of being a lesion in a colorectalor colonoscopic path.

FIG. 6 is a view showing a colonoscopic image diagnosis assisting systemand a workflow using the same according to an embodiment of the presentinvention.

When the diagnosis results and display information of the presentinvention are used in a hospital, they are displayed by the user systemhaving a user interface capable of displaying auxiliary artificialintelligence diagnosis results after endoscopic data has been receivedand analyzed, and then the diagnosis results may be verified, thediagnosis results may be replaced, or the acceptance or rejection of thediagnosis results may be determined based on user input.

The processor in a diagnosis assisting system 610 may store the displayinformation in the database with the display information associated withthe endoscopic image frame. In this case, the database may be a databaseinside the diagnosis computing system, and may be stored as medicalrecords for a patient in the future.

The processor may generate external storage data in which the displayinformation and the endoscopic image frame are associated with eachother, and may transmit the external storage data to an externaldatabase via the transmission module so that the external storage datacan be stored in the external database. In this case, the externaldatabase may be a PACS database or a database implemented based on acloud.

In this case, a plurality of medical image diagnosis algorithms 612 and614 are artificial intelligence algorithms each using an artificialneural network, and the processor may generate evaluation scores basedon the respective diagnosis requirements/context-based diagnosisrequirements as descriptive information for the plurality of respectivemedical image diagnosis algorithms.

The diagnosis assisting system 610 of the present invention mayinternally include the at least two artificial intelligence diagnosisalgorithms 612 and 614. Endoscopic image data is transferred from threeor more pieces of endoscopy equipment to the diagnosis assisting system610. The diagnosis assisting system 610 generates diagnosis results byapplying the at least two artificial intelligence diagnosis algorithmsto each frame of the endoscopic image data. The diagnosis assistingsystem 610 generates display information by associating the diagnosisresults with the frame of the endoscopic image data. In this case, thedisplay information may be generated to include the identificationinformation of a hospital (hospital A) in which the endoscopic imagedata is generated. Furthermore, the display information may be generatedto include the identification information (endoscope 1, endoscope 2, orendoscope 3) given to each piece of endoscope equipment of eachhospital.

FIG. 7 is a view showing a colonoscopic image diagnosis assisting systemand a workflow using the same according to an embodiment of the presentinvention.

The diagnosis assisting system 610 of FIG. 6 transmits the generateddisplay information to a cloud-based database, and the endoscopic imagedata and the display information are stored in the cloud-based databasein the state in which the endoscopy equipment, in which the endoscopicimage data was generated, and the hospital, in which the endoscopicimage data was generated, are identified. The display information may begenerated by associating diagnosis information with each frame of theendoscopic image data and then stored. The diagnosis informationgenerated for each frame of the endoscopic image data may beautomatically generated based on evaluation scores and context-baseddiagnosis requirements, as described in the embodiments of FIGS. 1 to 5.

When the present invention is applied in a cloud environment, endoscopicimage data and diagnosis results may be received by a user system on ahospital side using equipment connected over a wireless communicationnetwork, and auxiliary artificial intelligence diagnosis results may bedisplayed on the user system.

The display information stored in the cloud database may be provided toa hospital designated by a patient, and the patient may receive his orher endoscopic image data and diagnosis information at a hospital thatis convenient to access and also receive a doctor's interpretation ofdiagnosis results and a follow-up diagnosis at the hospital.

In the embodiments of FIGS. 1 to 7, the real-time image acquisitionmodule acquires a real-time endoscopic image from the endoscopic imagediagnosis acquisition module/endoscopic equipment. The real-time imageacquisition module transmits the real-time endoscopic image to thediagnosis assisting system. The diagnosis assisting system includes atleast two artificial intelligence algorithms, and generates displayinformation including diagnosis information by applying the at least twoartificial intelligence algorithms to the real-time endoscopic image.The diagnosis assisting system transfers the display information to theuser system, and the user system may overlay the display information onthe real-time endoscopic image or display the real-time endoscopic imageand the display information together.

The real-time endoscopic image may be divided into individual imageframes. In this case, the endoscopic image frames may be received by thereceiving interface.

The diagnosis assisting system (a computing system) includes thereception interface module, the processor, the transmission interfacemodule, and the memory/storage. The processor includes sub-modules thefunctions of which are internally implemented by hardware or software.The processor may include a first sub-module configured to extractcontext-based diagnosis requirements, a second sub-module configured toselect artificial intelligence analysis results to be displayed fromamong diagnosis results generated by applying artificial intelligencediagnosis algorithms to the endoscopic image frame, and a thirdsub-module configured to generate the display information to bedisplayed on the screen of the user system.

The plurality of artificial intelligence diagnosis algorithms may bestored in the memory or database (not shown) inside the diagnosiscomputing system, may be applied to the endoscopic image frame under thecontrol of the processor, and may generate diagnosis results for theendoscopic image frame.

Although a case where the plurality of artificial intelligence diagnosisalgorithms is stored in the memory or database (not shown) inside thediagnosis computing system and run under the control of the processor isdescribed in the embodiments of FIGS. 1 to 7, the plurality ofartificial intelligence diagnosis algorithms may be stored in memory ora database (not shown) outside the diagnosis computing system accordingto another embodiment of the present invention. When the plurality ofartificial intelligence diagnosis algorithms is stored in the memory ordatabase (not shown) outside the diagnosis computing system, theprocessor may control the memory or database (not shown) outside thediagnosis computing system via the transmission module so that theplurality of artificial intelligence diagnosis algorithms is applied tothe endoscopic image frame and diagnosis results for the endoscopicimage frame are generated. In this case, the generated diagnosis resultsmay be transferred to the diagnosis computing system through thereceiving interface, and the processor may generate the displayinformation based on the diagnosis results.

The processor extracts diagnosis requirements for the endoscopic imageframe by analyzing the endoscopic image frame, which is an image frameof a medical image. The processor selects a plurality of diagnosisapplication algorithms to be applied to the diagnosis of the endoscopicimage frame from among the plurality of medical image diagnosisalgorithms based on the diagnosis requirements, and the processorgenerates the display information including diagnosis results for theendoscopic image frame by applying the plurality of diagnosisapplication algorithms to the endoscopic image frame. This process isperformed on each of the endoscopic image frames by the processor.

The processor may extract context-based diagnosis requirementscorresponding to the characteristics of the endoscopic image frame byanalyzing the endoscopic image frame. The processor may select aplurality of diagnosis application algorithms to be applied to thediagnosis of the endoscopic image frame based on the context-baseddiagnosis requirements.

The processor may select a combination of a plurality of diagnosisapplication algorithms based on the context-based diagnosisrequirements. The processor may generate the display informationincluding diagnosis results for the endoscopic image frame by applyingthe plurality of diagnosis application algorithms to the endoscopicimage frame.

The combination of a plurality of diagnosis application algorithms mayinclude a first diagnosis application algorithm configured to bepreferentially recommended for the endoscopic image frame based oncontext-based diagnosis requirements, and a second diagnosis applicationalgorithm configured to be recommended based on a supplemental diagnosisrequirement derived from the context substrate diagnosis requirementsbased on a characteristic of the first diagnosis application algorithm.

The context-based diagnosis requirements may include one or more of abody part of the human body included in the endoscopic image frame, anorgan of the human body, a relative location indicated by the endoscopicimage frame in the organ of the human body, the probabilities ofoccurrence of lesions related to the endoscopic image frame, the levelsof risk of the lesions related to the endoscopic image frame, the levelsof difficulty of identification of the lesions related to the endoscopicimage frame, and the types of target lesions.

When an organ to which the endoscopic image frame is directed isspecified, for example, when the endoscopic image frame is related to acolonoscopic image, information about whether the image displayed in thecurrent image frame is the beginning, middle, or end of the colonoscopicimage may be identified along with the relative location thereof in thecolon (the inlet, middle, and end of the organ). In the case of acolonoscopic image, information about whether the image displayed in thecurrent image frame is the beginning (e.g., the esophagus), middle (theinlet of the colon), or end of the colonoscopic image may be identifiedalong with the relative location thereof in a colonoscopic path.

Accordingly, the context-based diagnosis requirements may be extractedbased on the types of lesions/diseases that are likely to occur at theidentified location and region, the types of lesions/diseases that arelikely to be overlooked by medical staff because they are difficult toidentify with the naked eye, diagnosis information aboutlesions/diseases that are not easy to visually identify within thecurrent image frame, and the types of lesions/diseases requiringattention due to their high risk/lethality during a diagnosis among thelesions/diseases that may occur at locations within the organ of thehuman body to which the current image frame is directed. In this case,the context-based diagnosis requirements may also include informationabout the types of target lesions/diseases that need to be firstconsidered in relation to the current image frame based on theinformation described above.

The display information may include the endoscopic image frame, thediagnosis results selectively overlaid on the endoscopic image frame,information about the diagnosis application algorithms having generatedthe diagnosis results, and evaluation scores for the diagnosisapplication algorithms.

Although priorities may be allocated to the artificial intelligencediagnosis algorithm in descending order of evaluation scores in theapplication of diagnoses, there are some additional factors to be takeninto consideration.

When a first-priority artificial intelligence algorithm detects a partof the lesions that are likely to occur in connection with thecorresponding endoscopic image and a subsequent-priority artificialintelligence algorithm detects an item that is not detected by the firstpriority algorithm, both the diagnosis results of the first-priorityartificial intelligence algorithm and the diagnosis results of thesubsequent-priority artificial intelligence algorithm may be displayedtogether. Furthermore, there may be provided a menu that allows a userto select a final diagnosis application artificial intelligencealgorithm based on the above-described criteria. In order to help theuser to make a selection, the menu may be displayed together with thediagnosis results of the plurality of AI algorithms and a description ofthe reason for displaying the diagnosis results.

For example, it is assumed that lesions A1 and A2 are known as being thetypes of lesions that are most likely to occur within the current imageframe and a lesion B is known as being less likely to occur than lesionsA1 and A2 and being likely to be overlooked because it is difficult tovisually identify. An artificial intelligence diagnosis algorithm X,which has obtained the highest evaluation score for the lesions A1 andA2, may obtain the highest overall evaluation score and be selected asthe first diagnosis application algorithm that is preferentiallyrecommended. Meanwhile, there may be a case where the first diagnosisapplication algorithm obtains the highest evaluation score for thelesions A1 and A2 but obtains an evaluation score less than a referencevalue for the lesion B. In this case, the lesion B for which the firstdiagnosis application algorithm exhibits the performance less than thereference value may be designated as a supplemental diagnosisrequirement. An artificial intelligence diagnosis algorithm. Y thatobtains the highest evaluation score for the lesion B, which is asupplemental diagnosis requirement, may be selected as a seconddiagnosis application algorithm. A combination of the first and seconddiagnosis application algorithms may be selected such that thecombination has high evaluation scores for the reliability and accuracyof the overall diagnostic information, the diagnostic information for aspecific lesion/disease is prevented from being overlooked, and thediagnostic performance for a specific lesion/disease is prevented frombeing poor. Accordingly, logical conditions for the selection of adiagnosis application algorithm may be designed such that an artificialintelligence diagnosis algorithm exhibiting the best performance for thesupplemental diagnosis requirement for which the first diagnosisapplication algorithm is weak, rather than the AI diagnosis algorithmexhibiting a high overall evaluation score, is selected as the seconddiagnosis application algorithm.

Although the case where the two diagnosis application algorithms areselected has been described as an example in the above embodiment, anembodiment in which three or more diagnosis application algorithms areselected and applied may also be implemented according to thedescription given herein in the case where the combination of the threeor more diagnosis application algorithms exhibits better performanceaccording to the evaluation scores.

The embodiments of FIGS. 1 to 7 are embodiments in which the diagnosisresults obtained by the application of the artificial intelligencediagnosis algorithms having high internal evaluation scores arepresented and then a user may select the diagnosis results obtained bythe application of artificial intelligence diagnosis algorithms havinghigher evaluation scores. In the embodiment of FIGS. 1 to 7, there isdisclosed a configuration conceived for the purpose of rapidlydisplaying diagnosis results for a real-time endoscopic image.Accordingly, in the embodiment of FIGS. 1 to 7, a combination ofartificial intelligence diagnosis algorithms to be displayed for thecurrent image frame is preferentially selected based on context-baseddiagnosis requirements, the diagnosis results of this combination aregenerated as display information, and the display information togetherwith the image frame is provided to a user.

In this case, the types of lesions/diseases that are likely to occur inthe current image frame, the types of lesions/diseases that are likelyto occur in the current image frame and are also likely to be overlookedby medical staff because they are difficult to visually identify, andthe types of lesions/diseases requiring attention during diagnosis dueto their high risk/lethality among the lesions that may occur in thecurrent image frame may be included in the context-based diagnosisrequirements. Furthermore, the types of target lesions/diseases thatshould not be overlooked in the current image frame based on the typesand characteristics of lesions/diseases, and the priorities of the typesof target lesions/diseases may be included in the context-baseddiagnosis requirements.

Diagnosis results are generated using results, to which artificialintelligence algorithms are applied, by the diagnosis computing terminalof the medical staff. In this case, the comments of the medical staffmay be added during the process of generating diagnosis results.

In the medical image diagnosis assisting system according to the presentinvention, the I-scores, i.e., evaluation scores, are transferred fromthe computing system to the diagnosis computing terminal of the medicalstaff. Final diagnosis texts may be generated by incorporating theI-scores, i.e., evaluation scores, into the generation of the diagnosisresults. According to an embodiment of the present invention, thecomputing system may generate diagnosis texts together with theI-scores, i.e., evaluation scores, and transfer them to the computingsystem of the medical staff. In this case, the diagnosis texts generatedby the computing system may be written using the diagnosis results basedon diagnosis application algorithms having higher I-scores, i.e., higherevaluation scores.

The computing system may provide a user interface configured to allowrecommended diagnosis results to be selected using I-scores, i.e.,internally calculated evaluation scores, and to allow a radiologist toevaluate/check diagnostic confidence in corresponding recommendeddiagnoses (e.g., recommended diagnosis algorithms consistent with thediagnosis results of the radiologist) because the evaluation scores arealso displayed. The processor of the computing system may select thefirst and second diagnosis results from among the plurality of diagnosisresults as recommended diagnosis results based on the evaluation scores.The processor may generate display information, including the evaluationscore for the first diagnosis algorithm, the first diagnosis result, theevaluation score for the second diagnosis algorithm, and the seconddiagnosis result.

The computing system may generate an evaluation score based on theconfidence score of a corresponding diagnosis algorithm, the accuracyscore of the diagnosis algorithm, and the evaluation confidence score ofa radiologist who provides feedback. The processor may generate theconfidence score of each of the plurality of medical image diagnosisalgorithms, the accuracy score of the medical image diagnosis algorithm,and the evaluation confidence score of the medical image diagnosisalgorithm by the user as sub-evaluation items based on a correspondingone of the plurality of diagnosis results and feedback on the diagnosisresult, and may generate an evaluation score based on the sub-evaluationitems.

For example, the criteria for the generation of the evaluation score maybe implemented as follows:

I-score=a×(the confidence score of an artificial intelligencealgorithm)+b×(the accuracy score of the artificial intelligencealgorithm)+c×(the evaluation confidence score of the artificialintelligence algorithm by a radiologist)  (1)

The confidence score of the algorithm may be given to the algorithm bythe radiologist. In other words, when it is determined that the firstdiagnosis result is more accurate than the second diagnosis result, ahigher confidence score may be given to the first diagnosis result.

The accuracy score of the algorithm may be determined based on theextent to which the radiologist accepts the diagnosis result of thealgorithm without a separate score giving process. For example, in thecase where when the first diagnosis result presents ten suspected lesionlocations, the radiologist approves nine suspected lesion locations, theaccuracy score may be given as 90/100.

Another embodiment in which the accuracy score of the algorithm is givenmay be a case where an accurate result is revealed through a biopsy orthe like. In this case, the accuracy of the diagnosis result of thediagnosis algorithm may be revealed in comparison with the accurateresult obtained through the biopsy. When the user inputs the accurateresult, obtained through the biopsy, to the computing system, thecomputing system may calculate the accuracy score of the diagnosisalgorithm by comparing the diagnosis result with the accurate resultobtained through the biopsy (a reference).

The evaluation confidence score of the radiologist may be provided as aconfidence score for the evaluation of the radiologist. In other words,when the radiologist is an expert having a loner experience in acorresponding clinical field, a higher evaluation confidence score maybe given accordingly. The evaluation confidence score may be calculatedby taking into consideration the years of experience of the radiologist,the specialty of the radiologist, whether or not the radiologist is amedical specialist, and the experience in the corresponding clinicalfield.

The computing system may update evaluation score calculation criteriaaccording to a predetermined internal schedule while continuouslylearning the evaluation score calculation criteria by means of aninternal artificial intelligence algorithm. The processor may assignweights to the confidence scores of the plurality of respective medicalimage diagnosis algorithms, the accuracy scores of the plurality ofrespective medical image diagnosis algorithms, and the evaluationconfidence scores of the plurality of respective medical image diagnosisalgorithms by the user, which are sub-evaluation items, and may updatethe weights of the sub-evaluation items so that the weights of thesub-evaluation items can be adjusted according to a target requirementbased on the plurality of diagnosis results and feedback on theplurality of diagnosis results by the user.

An example of the target requirement may be a case where adjustment isperformed such that there is a correlation between the confidence of theuser in the algorithms and the accuracy of the algorithms. For example,first and second diagnosis algorithms having the same accuracy score mayhave different confidence scores that are given by a radiologist. Inthis case, when confidence scores are different from each other whileexhibiting a certain tendency after the removal of the general errors ofthe evaluation of the radiologist, it can be recognized that theconfidence of the radiologist in the first diagnosis algorithm isdifferent from the confidence of the radiologist in the second diagnosisalgorithm. For example, in the case where the first and second diagnosisalgorithms generate accurate diagnosis results at nine of a total of tensuspected lesion locations, resulting in an accuracy score of 90/100 butonly the first diagnosis algorithm accurately identifies a severe lesionand the second diagnosis algorithm does not identify the lesion, theconfidence of the radiologist in the first diagnosis algorithm may bedifferent from the confidence of the radiologist in the second diagnosisalgorithm. A means for adjusting the correlation between the accuracyand the confidence may be a means for adjusting the weights of therespective sub-evaluation items or subdividing criteria for theselection of target lesions related to the determination of accuracy. Inthis case, there may be used a method that classifies lesions accordingto criteria such as the hardness/severity of an identified lesion, theposition of the lesion from the center of a medical image, and thedifficulty of identifying the lesion (the difficulty is high in a regionwhere bones, organs, and blood vessels are mixed in a complicated form)and assigns different weights to the diagnosis accuracies of lesions inrespective regions.

The computing system may include a function of automatically allocatinga plurality of artificial intelligence algorithms that are applicabledepending on an image. To determine a plurality of artificialintelligence algorithms applicable to an image, the computing system 100may classify one examination or at least one image by means of aseparate image classification artificial intelligence algorithm inside arecommendation diagnosis system, and may then apply a plurality ofartificial intelligence algorithms.

In an embodiment of the present invention, the plurality of medicalimage diagnosis algorithms may be medical image diagnosis algorithmsusing artificial neural networks. In this case, the evaluation score andthe sub-evaluation items may be generated as descriptive information foreach diagnosis algorithm, and the computing system 100 may feed theevaluation score and the sub-evaluation items back to the creator of thediagnosis algorithm so that the information can be used to improve thediagnosis algorithm.

In this case, when each of the artificial neural networks is anartificial neural network using a relevance score and a confidencelevel, which is being studied recently, a statistical analysis isperformed with the evaluation score and the sub-evaluation itemsassociated with the relevance score or confidence level of theartificial neural network, and thus the evaluation score and thesub-evaluation items may affect the improvement of the diagnosisalgorithm.

This embodiment of the present invention is designed to provideadvantages obtainable by the present invention while minimizing thedeformation of the medical image diagnosis sequence of the related artas much as possible.

In another embodiment of the present invention, the computing system mayperform the process of generating a plurality of diagnosis results byselecting a plurality of diagnosis application algorithms and thenapplying the plurality of diagnosis application algorithms to a medicalimage by itself. In this case, the computing system may transfer notonly information about the selected diagnosis application algorithms butalso the plurality of diagnosis results based on the diagnosisapplication algorithms to the diagnosis computing terminal of themedical staff, and the results obtained by applying artificialintelligence algorithms (the diagnosis application algorithms) to themedical image may be displayed on the diagnosis computing terminal ofthe medical staff.

In this case, an embodiment of the present invention may provideadvantages obtainable by present invention even when the computing powerof the diagnosis computing terminal of the medical staff is not high,e.g., the diagnosis computing terminal of the medical staff is a mobiledevice or an old-fashioned computing system. In this case, in anembodiment of the present invention, an agent that applies theartificial intelligence algorithms to the medical image is the computingsystem, the computing system functions as a type of server, and thediagnosis computing terminal of the medical staff may operate based on athin-client concept. In this case, in an embodiment of the presentinvention, the feedback indicators input for the plurality of diagnosisresults or the plurality of diagnosis application algorithms via thediagnosis computing terminal of the medical staff by the medical staffmay be fed back to the computing system. The feedback indicators may bestored in the memory or database inside the computing system inassociation with the evaluation targets, i.e., the plurality ofdiagnosis results or the plurality of diagnosis application algorithms.

As described above, in an embodiment of the present invention, the stepof applying the selected algorithms may be performed in the diagnosissystem of the clinician, and a plurality of diagnosis results may betransferred to the computing system. In another embodiment of thepresent invention, the overall step of applying the selected algorithmsmay be performed within the computing system and then the results of theapplication may be displayed on the diagnosis system of the clinician.

According to the present invention, work efficiency and diagnosticaccuracy may be increased by training on polyps, etc., which may bemissed by a user, based on artificial intelligence medical imagediagnosis results for each real-time video frame of a colonoscopic imagevia the artificial intelligence algorithm and then applying the resultsof the training to the artificial intelligence diagnosis assistingsystem.

According to the present invention, there is an effect of preventing inadvance a situation that may develop into cancer by detecting a lesionor the like at its early stage. Not only lesions of various sizes butalso the locations of the lesions in colonoscopic paths are included inlabels and used as learning data. Accordingly, according to the presentinvention, lesion detection rate may be increased by automaticallydetecting even a considerably small lesion that may easily be missed bya user, and also the locations of lesions in colonoscopic paths may beextracted.

According to the present invention, the incidence of colorectal cancermay be reduced by increasing polyp/lesion detection rate and alsoeliminating colorectal cancer risk factors in their early stages.Furthermore, the contributions may be made to reducing the causes ofcolorectal cancer and reducing the frequency of examinations by enablingdoctors to find and treat more lesions than before.

According to the present invention, a disease that may easily be missedby a user may be automatically detected during colonoscopy and thelocation of the disease in a colorectal path (a colonoscopic path) maybe presented, so that the user may easily check the disease in real timeduring colonoscopy and even a report adapted to enable other examinersto check it later may be generated through a simple operation.

According to the present invention, there may be provided the optimizedcontent of artificial intelligence medical image diagnosis results foreach real-time image frame of an endoscopic image.

According to the present invention, there may be provided the optimizedcontent of a plurality of artificial intelligence medical imagediagnosis results for each real-time image frame.

According to the present invention, there may be provided an optimizedcombination of a plurality of artificial intelligence medical imagediagnosis results as display information for each real-time image frame.

According to the present invention, there may be provided an optimizedcombination of a plurality of artificial intelligence medical imagediagnosis results capable of efficiently displaying diagnosis resultsthat are likely to be acquired, are likely to be overlooked, or have ahigh level of risk in a current image frame.

According to the present invention, there may be provided the userinterface and diagnosis computing system that automatically detect andpresent diagnosis results that are likely to be acquired, are likely tobe overlooked, or have a high level of risk in a current image frame, sothat medical staff can check and review the diagnosis results in realtime during an endoscopy.

A medical image diagnosis assisting method according to the embodimentsof the present invention may be implemented in the form of programinstructions, and may be then stored in a computer-readable storagemedium. The computer-readable storage medium may include programinstructions, data files, and data structures solely or in combination.Program instructions recorded on the storage medium may have beenspecially designed and configured for the present invention, or may beknown to or available to those who have ordinary knowledge in the fieldof computer software. Examples of the computer-readable storage mediuminclude all types of hardware devices specially configured to record andexecute program instructions, such as magnetic media, such as a harddisk, a floppy disk, and magnetic tape, optical media, such as compactdisk (CD)-read only memory (ROM) and a digital versatile disk (DVD),magneto-optical media, such as a floptical disk, ROM, random accessmemory (RAM), and flash memory. Examples of the program instructionsinclude machine code, such as code created by a compiler, and high-levellanguage code executable by a computer using an interpreter. Thesehardware devices may be configured to operate as one or more softwaremodules in order to perform the operation of the present invention, andthe vice versa.

However, the present invention is not limited to the embodiments. Likereference symbols in the drawings designate like components. Thelengths, heights, sizes, widths, etc. introduced in the embodiments anddrawings of the present invention may be exaggerated to help tounderstand.

Although the present invention has been described with reference tospecific details such as the specific components, and the limitedembodiments and drawings, these are provided merely to help a generalunderstanding of the present invention, and the present invention is notlimited thereto. Furthermore, those having ordinary skill in thetechnical field to which the present invention pertains may make variousmodifications and variations from the above detailed description.

Therefore, the spirit of the present invention should not be definedbased only on the described embodiments, and not only the attachedclaims but also all equivalent to the claims should be construed asfalling within the scope of the spirit of the present invention.

What is claimed is:
 1. A colonoscopic image diagnosis assisting systemfor assisting diagnosis of a medical image, the colonoscopic imagediagnosis assisting system comprising a computing system, wherein thecomputing system comprises: a receiving interface configured to receivea colonoscopic image as the medical image; memory or a databaseconfigured to store at least one medical image analysis algorithm havinga function of analyzing the colonoscopic image; a processor; and a userdisplay, wherein the processor is configured to: analyze each videoframe of the colonoscopic image using the at least one medical imageanalysis algorithm; detect a finding suspected of being a lesion in thevideo frame; calculate coordinates of a location of the findingsuspected of being a lesion; and generate display information, includingwhether the finding suspected of being a lesion is present and thecoordinates of the location of the finding suspected of being a lesion,and wherein the user display is configured to display the findingsuspected of being a lesion so that it is visually distinguished in thevideo frame based on the display information and display the coordinatesof the location of the finding suspected of being a lesion so that theyare visually associated with the finding suspected of being a lesion. 2.The colonoscopic image diagnosis assisting system of claim 1, whereinthe processor is further configured to calculate a location of thefinding suspected of being a lesion in a colonoscopic path, wherein theprocessor is further configured to generate display information,including whether the finding suspected of being a lesion is present,the coordinates of the location of the finding suspected of being alesion, and the location of the finding suspected of being a lesion inthe colonoscopic path, and wherein the user display is furtherconfigured to display the location of the finding suspected of being alesion in the colonoscopic path so that it is visually associated withthe finding suspected of being a lesion based on the displayinformation.
 3. The colonoscopic image diagnosis assisting system ofclaim 2, wherein the processor is further configured to track a locationof the video frame, indicative of a current examination region, in thecolonoscopic path, and wherein the processor is further configured tocalculate the location of the finding suspected of being a lesion in thecolonoscopic path based on the location of the video frame in thecolonoscopic path and the coordinates of the location of the findingsuspected of being a lesion.
 4. The colonoscopic image diagnosisassisting system of claim 2, wherein the processor is further configuredto calculate the location of the finding suspected of being a lesion inthe colonoscopic path based on a pre-examination medical image includinga three-dimensional anatomical structure of a patient to be examined. 5.The colonoscopic image diagnosis assisting system of claim 1, whereinthe processor is further configured to extract context-based diagnosisrequirements corresponding to characteristics of the video frame in themedical image by analyzing the video frame of the medical image, andwherein the processor is further configured to select a diagnosisapplication algorithm to perform diagnosis on the video frame from amonga plurality of diagnosis application algorithm candidates based on thecontext-based diagnosis requirements.
 6. The colonoscopic imagediagnosis assisting system of claim 1, wherein the receiving interfaceis further configured to receive at least one colonoscopic image from atleast one colonoscopic image acquisition module, wherein the processoris further configured to detect a finding suspected of being a lesionfor each video frame of the at least one colonoscopic image by using theat least one medical image analysis algorithm, and wherein the processoris further configured to generate display information, including whetherthe finding suspected of being a lesion is present and the coordinatesof the location of the finding suspected of being a lesion, for eachvideo frame of the at least one colonoscopic image.
 7. A colonoscopicimage diagnosis assisting method, the method being performed by acolonoscopic image diagnosis assisting system including a processor anda user display, the method comprising: receiving a colonoscopic image;analyzing, by the processor, each video frame of the colonoscopic imageby using at least one medical image analysis algorithm having a functionof analyzing the colonoscopic image stored in memory or a database inthe colonoscopic image diagnosis assisting system, and detecting, by theprocessor, a finding suspected of being a lesion in the video frame;when the finding suspected of being a lesion is present in the videoframe, calculating, by the processor, coordinates of a location of thefinding suspected of being a lesion; generating, by the processor,display information, including whether the finding suspected of being alesion is present and the coordinates of the location of the findingsuspected of being a lesion; when the finding suspected of being alesion is present in the video frame, displaying, by the user display,the finding suspected of being a lesion so that it is visuallydistinguished in the video frame based on the display information; anddisplaying, by the user display, the coordinates of the location of thefinding suspected of being a lesion so that they are visually associatedwith the finding suspected of being a lesion.
 8. The colonoscopic imagediagnosis assisting method of claim 7, further comprising calculating,by the processor, a location of the finding suspected of being a lesionin a colonoscopic path, wherein the generating comprises generating, bythe processor, display information, including whether the findingsuspected of being a lesion is present, the coordinates of the locationof the finding suspected of being a lesion, and the location of thefinding suspected of being a lesion in the colonoscopic path, andwherein the displaying the coordinates of the location of the findingsuspected of being a lesion comprises displaying, by the user display,the location of the finding suspected of being a lesion in thecolonoscopic path so that it is visually associated with the findingsuspected of being a lesion based on the display information.
 9. Thecolonoscopic image diagnosis assisting method of claim 7, wherein thereceiving comprises receiving at least one colonoscopic image from atleast one colonoscopic image acquisition module, wherein the detectingcomprises detecting whether a finding suspected of being a lesion ispresent for each video frame of the at least one colonoscopic image byusing the at least one medical image analysis algorithm, and wherein thegenerating comprises generating display information, including whetherthe finding suspected of being a lesion is present and the coordinatesof the location of the finding suspected of being a lesion, for eachvideo frame of the at least one colonoscopic image.